Spatially-correlated multi-display human-machine interface

ABSTRACT

A human-machine interface involves plural spatially-coherent visual presentation surfaces at least some of which are movable by a person. Plural windows or portholes into a virtual space, at least some of which are handheld and movable, are provided by using handheld and other display devices. Aspects of multi-dimensional spatiality of the moveable window (e.g., relative to another window) are determined and used to generate images. As one example, the moveable window can present a first person perspective “porthole” view into the virtual space, this porthole view changing based on aspects of the moveable window&#39;s spatiality in multi-dimensional space relative to a stationary window. A display can present an image of a virtual space, and an additional, moveable display can present an additional image of the same virtual space.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a division of U.S. patent application Ser. No.13/153,106 filed on Jun. 3, 2011, which application is acontinuation-in-part of U.S. patent application Ser. No. 13/019924 and acontinuation-in-part of U.S. patent application Ser. No. 13/019928, eachfiled on Feb. 2, 2011, and each of which claims benefit under 35 U.S.C.Section 119 of the following Japanese Patent Applications: 2010-022022and 2010-022023 filed Feb. 3, 2010; 2010-177893 filed Aug. 6, 2010;2010-185315 filed Aug. 20, 2010; 2010-192220 and 2010-192221 filed Aug.30, 2010; and 2010-245298 and 2010-245299 filed Nov. 1, 2010. Thecontents of all of these US and Japanese applications are incorporatedby reference herein in their entirety.

FIELD

The technology herein relates to human-machine interfaces, and moreparticularly to interactive graphical computer interfaces with multipledisplay surfaces. Still more particularly, the technology herein relatesto immersive, intuitive, interactive multi-display human-machineinterfaces offering plural spatially-correlated displays (e.g., handheldand other) to enhance the user experience through virtual spatiality.

BACKGROUND AND SUMMARY

We humans exist in three-dimensional space. We know where we are withinthe 3D world that surrounds us. Turn your head to the left. Now turn itto the right. Look up. Look down. You have just determined your placewithin the 3D environment of the room.

Ever since computer graphics became interactive, people have been tryingto create human-machine interfaces that provide the same sort ofimmersive spatial experience we get from looking around the real world.As one example, systems are known that provide users withinertially-sensed head-mounted displays that allow them to look aroundand interact with a so-called CAVE virtual environment. See e.g., Buxtonet al, HMD's, Caves & Chameleon: A Human-Centric Analysis of Interactionin Virtual Space, Computer Graphics: The SIGGRAPH Quarterly, 32(4),64-68 (1998). However, some kinds of head mounted displays can restrictthe user's field of view or otherwise impair a person's ability tointeract with the real world.

Flat screen televisions and other such displays common in many homestoday can be used to display a virtual world. They provide a reasonablyimmersive environment with high resolution or high definition at lowcost, and most consumers already own one. Some are even “3D” (e.g.,through use of special glasses) to provide apparent image depth. Theperson watching or otherwise interacting with a television or other suchdisplay typically at least generally faces it and can see and interactwith the virtual environment it presents. Other people in the room canalso see and may be able to interact. These displays are wonderful fordisplaying immersive content but, unlike stereo or theater sound, theygenerally do not wrap around or envelop the user. Wrap-around projectedor other displays are known (see e.g., Blanke et al, “ActiveVisualization in a Multidisplay Immersive Environment”, EighthEurographics Workshop on Virtual Environments (2002)), but may be tooexpensive for home use. Further innovations are possible.

Some non-limiting example implementations of technology herein provide amovable (e.g., handholdable) window or porthole display surface into avirtual space. Aspects of multi-dimensional spatiality of the moveabledisplay surface relative to another e.g., stationary display aredetermined and used to generate images while at least some spatialaspects of the plural displays are correlated. As one non-limitingexample, the moveable display can present a first person perspective“porthole” view into the virtual space, this porthole view changingbased on aspects of the moveable display's spatiality inmulti-dimensional space relative to a stationary display providing acontextual spatial reference for the user.

In one non-limiting aspect, a human-machine interface involves pluralvisual presentation surfaces at least some of which are movable. Forexample, a display can present an image of a virtual space, and another,moveable display can present an additional image of the same virtualspace but from a potentially different viewpoint, viewing perspective,field of view, scale, image orientation, augmentation and/or other imagecharacteristic or aspect.

The non-limiting movable display can selectively display an additionalimage of the virtual space and/or other user interface information(e.g., text, graphics or video relating or auxiliary to the images theother display presents) depending on the movable display's attituderelative to the other display.

In some example implementations, the movable display can act under somecircumstances as a pointing device. When pointed/aimed at anotherdisplay, the movable display's attitude can control or influence theposition of a pointing symbol on the other display. When pointed/aimedaway from the other display, the movable display can display a furtherimage of the virtual space displayed on the other display, but from adifferent viewing direction that depends on the movable display'sattitude.

Technology is used to determine aspects of the spatiality of at leastone of the display devices in the physical world, and to use thedetermined spatiality aspects to present appropriately-viewpointed,-viewing-perspectived, -directioned and/or other characteristic imageson the displays. As one non-limiting example, determined spatiality ofthe movable display relative to a stationary or other display can beused to provide relative spatiality of images the plural displayspresent.

One example non-limiting implementation provides an immersive spatialhuman-machine interface comprising at least one handheld display movablein free space; an arrangement configured to determine at least someaspects of the attitude of the movable handheld display; at least oneadditional display; and at least one graphical image generatoroperatively coupled to the handheld display and the additional display,the at least one graphical image generator generating images of avirtual space for display by each of the handheld display and theadditional display, wherein the at least one graphical image generatorgenerates images from different viewing directions images of differentperspectives (viewing) from a same or similar virtual location, viewingpoint, neighborhood, vantage point, neighborhood, vicinity, region orthe like, for display by the handheld display and the additional displayat least in part in response to the determined attitude aspects toprovide spatial correlation between the two images and thereby enhancethe immersive spatiality of the human-machine interface.

Such a non-limiting example interface may further provide that when twoimages are practically similar, the image presented by the handhelddisplay is substituted by other image(s) and/or associated information.In this context, “practically similar” may include or mean similar orthe same or nearly the same from the perception of the user who isviewing the two displays.

In some implementations, the additional display comprises a relativelylarge stationary display, and the at least one graphical image generatordoes not practically alter the rendering perspective (or viewpoint),except perhaps some marginal, peripheral or border-situation look-aroundperspective shifts, for images generated for the stationary displaybased on the determined attitude aspects.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages will be better and morecompletely understood from the following detailed description ofexemplary non-limiting illustrative embodiments in conjunction with thedrawings, of which:

FIG. 1 shows an example non-limiting system including a movable displaydevice and a stationary display device;

FIG. 1A illustrates an example movement of an object (in this case anairplane) in six degrees of freedom;

FIGS. 2A & 2B show example images presented on the example movabledisplay device when the moveable display device translates vertically;

FIGS. 3A & 3B show example images presented on movable and stationarydisplay devices when the movable display device translates vertically;

FIGS. 4A & 4B show example images presented on the moveable displaydevice when the movable display device rotates about a roll axis;

FIGS. 5A & 5B show example images presented on movable and stationarydisplay devices when the movable display device rotates about a rollaxis;

FIG. 6 shows an example image presented on movable and stationarydisplay devices when the movable display device rotates about a yaw axisorthogonal to the roll axis;

FIG. 6A shows various example attitudes of the movable display devicerelative to the stationary display device including in-range,out-of-range and transition area relative attitudes;

FIGS. 6B through 6D show example non-limiting applications and usage foran in-range case;

FIGS. 6E through 6H show example non-limiting applications and usage foran out-of-range case;

FIGS. 6I through 6J show example non-limiting applications and usage fora transition area case;

FIG. 7 shows an example non-limiting application of the movable displaydevice;

FIG. 8A shows how a human can hold the movable display device in twohands;

FIG. 8B shows how a human can hold the movable display device in onehand and use the pointing finger of the other hand to point to the touchscreen;

FIG. 8C shows how the movable display device can be rested on a surface;

FIG. 8D shows how plural movable display devices can be used withtogether with a common stationary display;

FIG. 8E shows how a movable display device can be rested on a surface orheld and how a stylus can be used to point to and manipulate imagesdisplayed on the touch screen;

FIGS. 9A and 9B show example attachment of the moveable display deviceto an example gun type accessory;

FIGS. 10A-10D show more detailed views of an example non-limitingmovable display device;

FIG. 11 shows an example non-limiting block diagram of a movable displaydevice;

FIG. 12 shows an example non-limiting processing flowchart;

FIG. 12A shows a non-limiting example of a stationary display viewingfrustum and transformation, and a movable display viewing frustum andtransformation responsive to a movable display's current attitude;

FIG. 12B shows a further non-limiting processing flowchart;

FIGS. 13A and 13B show an example non-limiting virtual drivingsimulation user experience; and

FIGS. 14A and 14B show an example non-limiting virtual first personviewpoint user experience.

DETAILED DESCRIPTION OF NON-LIMITING EMBODIMENTS

FIG. 1 shows an example non-limiting system S that provides aspatially-immersive interactive multi-display human-machine interface.Example non-limiting system S includes two display devices SD, MD, onemovable (e.g., hand-holdable) and one stationary. The movable display MDin the example implementation is physically movable in free space duringuse, whereas the stationary display SD is relatively physicallystationary in space during use. Of course, stationary displays SD suchas home televisions can be moved from one place to another, but aregenerally not moved (or not moved much) during use. The relatively fixedposition of stationary display SD provides physical correlation of thephysical stationary display SD versus the movable display MD. Thus,“stationary” does not necessarily mean absolute immobility, but canencompass displays that are not generally moved (or not moved much)during use. In other non-limiting implementations, the stationarydisplay SD can also be movable or partially movable.

One or more graphics source(s) G control or enable display devices SD,MD to display computer-generated or other images. In the non-limitingexample shown, display devices SD, MD are spatially correlated todisplay respective aspects of a common virtual world for example fromdifferent viewpoints or viewing perspectives.

If display device SD is stationary, its location and attitude in 3Dspace can be assumed and need not be measured. Sensors T measure theattitude or other aspect(s) of potentially-changing spatiality ofmovable display device MD, e.g. using MARG (“Magnetic Angular RateGravity”) technology.

The graphics source(s) G are configured to be responsive to informationfrom the sensors T to control the images displayed on one or both ofdisplay devices SD, MD. For example, the graphics source(s) G mayinclude, or be in communication with, processing circuitry that receivesinformation from the sensors T and that controls the displayed imagesbased on the received sensor information.

In example non-limiting implementations, the perspectives, viewpointsand/or viewing directions of images displayed by movable display deviceMD respond to the spatiality of the movable display device. In someinstances, those image perspectives, viewing directions and/orviewpoints may be at least in part determined by apparent or actualrelative spatiality of the two display devices SD, MD to continuallyspatially-correlate the two display presentations. The series of imagespresented on movable display MD thus are correlated in both space andtime with the series images presented on stationary display SD in anexample non-limiting implementation.

In one particular non-limiting example, physical display device SD isphysically stationary in the room where it is located. It can forexample be a relatively large fixed display (e.g., a wall or tablemounted television or other display) used to present an image of a 3Dvirtual world—in this case a landscape including a breathtakingmountain. The images that display device SD displays can be static ordynamic. For example, they can be moving and/or animated (e.g., liveaction) images that dynamically change in various ways includingviewpoint, e.g., in response to user input or other factors.

In the non-limiting example shown, display device MD is movable in freespace. As movable display device MD is moved, it can display a movableimage of the same virtual world as is displayed on the stationarydisplay SD, but the image the movable display device displays of thevirtual world is transformed based on aspects of the spatiality ofmovable display device MD. For example, the attitude of the movabledisplay MD can be used to display the virtual world on the movabledisplay from a different viewpoint, viewing perspective, viewingdirection, field of view, image orientation, augmentation, scale and/orother image characteristic(s).

A human can move movable display device MD anywhere in free space.Moving an object anywhere in free space is sometimes called moving in“six degrees of freedom” (6DOF) because there are generally six basicways an object can move (see FIG. 1A showing an airplane as an example):

-   -   1) Up/down,    -   2) Left/right,    -   3) Forward/backward,    -   4) pitch rotation (nose up or down),    -   5) yaw rotation (compass direction or heading during level        flight), and    -   6) roll rotation (one or the other wing up).

For example, the person can move (“translate”) display device MD alongany of three orthogonal axes: up and down (vertical or “y”), left andright (horizontal or “x”), and forward and backward (depth or “z”). Theperson can rotate display device MD about any of three orthogonalrotational axes (pitch, yaw and roll). Just like any object, the personcan simultaneously move display device MD in any combination of these“six degrees of freedom” to move display device MD in any direction andmanner in three-dimensional space. For example, the person can spindisplay device MD about the roll (and/or any other) rotational axis atthe same time she translates the device down or in any otherdirection(s)—like an airplane that rolls and descends at the same time,for example.

In the example non-limiting implementation, system S has sensors T thatdetect aspects of the spatiality of movable display MD such as itsattitude. These sensors T may include one or more of accelerometers,gyrosensors, magnetometers, ultrasonic transducers, cameras, and thelike. These sensors T enable movable display device MD to displaycorresponding different parts of the virtual world (e.g., the mountain)from different viewpoints, viewing perspectives, viewing directions orother characteristics responsive to aspects of the display device'scurrent spatiality.

In one non-limiting example, the person can move display device MD tonew attitudes that permit the person to examine different parts of themountain. For example, to look at the base of the mountain, the personcan move or rotate display device MD downward. One or more of thesensors T detect this downward movement such as translation and/orrotation, and the graphics source(s) G are responsive to the downwardmovement detection to display the base of the mountain on the movabledisplay device MD. At the same time, the graphics source(s) G maycontinue to display the same view of the mountain on the stationarydisplay device, thereby effectively providing a view that the person canintuitively use as a reference or context to understand that the displaydevice MD is showing the base of the mountain and as a reference orcontext for further movement or rotation of display device MD to look atother parts of the mountain. For example, to look at the mountain'speak, the person can move or rotate the display device upwards.

In some non-limiting examples, the person can move display device MD toexamine parts of the 3D virtual world that can't currently be seen onstationary display SD. For example, the person can turn display deviceMD to the left or the right, up or down, or away from stationary displaydevice SD (e.g., so a ray projecting from the movable display device MDin a direction that is normal to the planar dimension of the movabledisplay device does not intersect the stationary display SD) tovisualize portions of the virtual world (e.g., other mountains, amountain lake, a valley, a town, other side(s) of the same mountain,etc.) that cannot currently be seen on stationary display SD.

Thus, in one particular example, the stationary display SD acts as aspatial reference that orients the user within the 3D virtual space. Itprovides a perceptual context or anchor for the person using movabledisplay device MD. Because the person can see both the stationarydisplay SD and the movable display MD at the same time, the person'sbrain takes both images into account. If at least some aspects of thetwo images are spatially correlated, the person's brain can interpolateor interpret the two displayed images to be two windows or other viewsinto the same virtual world. The movable display MD thus provides a 3Dspatial effect that is not available merely from the image displayed bythe stationary display SD.

The image displayed on stationary display SD can be static or dynamic.For example, it is possible to use a joystick to move a character in avirtual reality database or space. For example, the analog stick can beused to change the direction of the first person player and can controlthe walk of a first person through the game. An example could be avirtual reality walkthrough. In this case, the stationary (e.g., TV)display will display the character's view or location as a moving scene.Manipulating controls on the movable display MD (or changing theattitude of the movable display housing) could for example cause systemS to generate a series of images on stationary display SD, the imageschanging to show progression of a virtual character through the virtualscene. The changing images displayed on stationary display SD may thusbe animated in some example non-limiting implementation to exposedifferent objects, enemies, friends, and other virtual features as thevirtual character moves through the 3D virtual space. See FIG. 14A. Theuser can in some implementations control the viewpoint and viewingdirection of a virtual camera used to define the changing view in the 3Dspace. Once the virtual character stops and changes the attitude of themovable display MD is changed (see FIG. 14B), the user may have theoption to see additional view of the virtual reality database on themovable display MD. For example, the user can “look around” or “rubberneck” in the virtual space by changing the attitude of the movabledisplay MD to display views of the virtual space not currently displayedon the stationary display. It is possible for the virtual character tointeractively move (walk or run) at the same time as the person can usethe movable display MD to rubber neck or look around. While such lookingaround may perhaps most naturally occur once a virtual character hasstopped moving within the virtual environment, it is possible to walk orotherwise move and rubberneck at same time.

System S thus provides a high degree of interactivity. By interactingwith an additional, spatially-correlated viewing surface that the personcan move anywhere in 3D space, the person becomes immersed within thevirtual environment. She feels as if she is inside the virtualenvironment. She can look around, explore and discover new aspects ofthe virtual environment by rotating and/or translating the movabledisplay MD (or herself while she is holding the movable display device).It is even possible for the person to rubberneck or look inquisitivelyaround the virtual environment by looking about or survey withwonderment or curiosity. She can also interact with the virtualenvironment though additional user input arrangements on the movabledisplay MD. For example, the movable display MD provides, in onenon-limiting implementation, a touch screen TS that exposes a portion ofthe virtual environment to being directly manipulated (e.g., touched) bythe person. See FIG. 8B. Meanwhile, because the stationary display SDcontinues to be in view, it continually provides a reference orcontextual orienting anchor for the person as the person moves andfurther interacts with a handheld spatially-correlated patch of thevirtual environment.

For example, suppose the person is virtually moving in the virtualspace. The person can virtually walk through the virtual world (e.g., bycontrolling joystick or slider type controls on the movable display MD)and see corresponding changing images displayed on the stationarydisplay SD that reflect the user's progress and travels through thevirtual space. The person can at any time stop and look around. At thispoint, the system S can define two viewing angles from the approximatelysame location. In one example non-limiting implementation, the twodisplays (one on SD, the other on MD) can be based on the same orsimilar location but defined by two different viewing angles orperspectives.

In one non-limiting example, the size of movable display device touchscreen TS is small enough so that movable display device MD is easilymovable and holdable without fatigue, but large enough so thespatially-correlated patch of the 3D virtual world the movable displaydevice MD presents is significant in affecting the person's spatialperception. Because the movable display device MD is much closer to theperson's eyes than the stationary display SD, the person perceives it tobe larger in her visual field of view. In fact, the person can hold themovable display device MD such that it may temporarily block or obscureher view of stationary display SD. Nevertheless, the stationary displaySD remains present in the person's environment to be glanced at fromtime to time. It continues to anchor and orient the person's visualperception with a context that makes the spatially-correlated imagedisplayed by movable display MD appear to be part of the same virtualenvironment that the stationary display SD displays. Generally speaking,watching a larger screen on stationary display SD is more comfortablebecause it is big. If the same, practically the same or similar imagesare displayed on the stationary display SD and movable display MD, mostplayers will focus on the big screen rather than the small screen. Inmany non-limiting cases, a pointing cursor may be displayed on thestationary display SD as a pointing target. By changing the attitude ofthe movable display MD, it is possible to control the position of apointing target within the display or other presentation area of thestationary display SD.

In this context, “practically similar” or “practically the same” canmean the same or nearly the same from the user's perspective. In oneexample implementation, processing is not necessarily conditional on theimages being exactly the same. When the perspective is the same orsimilar from a user perspective, or when the situation is in range, thenthe processing will take place. In one non-limiting example, when “inrange” (e.g., when a ray projecting from the movable display device MDin a direction that is normal to the planar dimension of the movabledisplay device intersects the stationary display SD—or an assumedlocation of stationary display SD), the movable display will displaydifferent information. Thus, when the displays on the display deviceswould be essentially the same from the user's standpoint, the user caninstead see something different on the movable display MD.

For example, in another non-limiting example (see FIGS. 13A-13B), thestationary display SD may display a driver's view out of the frontwindshield of a car driving down a country road. In this FIG. 13Aexample, the movable display device MD can display “dashboardinformation” such as speed, fuel level, and other such information. Insome non-limiting examples, movable display device MD may display adimmed or grayed-out version of all (or some portion) of what isdisplayed on the stationary display SD and the dashboard information maybe overlaid onto the dimmed or grayed-out version. In other examples,the dashboard information may be overlaid onto some other background oron no background. Given this context or anchor presented on stationarydisplay SD, a person may intuitively move and/or rotate movable displaydevice MD to provide views of any passengers in the passenger's seat orback seat or the countryside as seen through the passenger's side ordriver's side window. The user can thus look out the window (FIG. 13B),look behind the virtual vehicle, etc. The person feels as if he isinside a car as the car is moving down the road. System S can generatesurround sound effects, tactile sensations or other perceptualinformation responsive to the spatiality that can add to this spatialimmersive experience.

In many non-limiting cases, the relatively small movable display MDshows status such as for example selection of weapons or other items orstatus information. It is not necessary for the subject to continuallywatch the movable display, but at any desired time the subject can touchto select items, change items, etc. If something interesting happens onthe big screen SD, e.g., sound and voices or some enemy flying up orflying away from the big screen, the person can change or move themovable display devices to a location that is “out of range” (e.g., whena ray projecting from the movable display device MD in a direction thatis normal to the planar dimension of the movable display device does notintersect the stationary display SD—or an assumed location of stationarydisplay SD), such as upwards, downwards or even backwards. A continuousarea surrounding the stationary display SD's view will be displayed onthe movable display MD, while keeping the stationary display in the sameor similar location(s). The bigger screen of stationary display SDbecomes an absolute or relative reference for the virtual world,allowing the person to turn back to or glance at the original place.

Illustrative Examples of Spatial Correlation Between The Displays

FIGS. 2A-7 illustrate example non-limiting spatial correlation betweenimages the movable display MD and stationary display SD present. Some ofthese Figures show movable display MD by itself (as if the stationarydisplay SD were not present or activated) in order to better explain howthe movable display is interacting with the virtual environment, andthen show spatial correlation between the movable and stationarydisplays.

FIGS. 2A-2B show movable display device MD acting as a window into asimple 3D virtual world or space comprising a series of polygons P1-P5.The display device MD and the person are in the real 3D world; thepolygons (P1-P5) shown in dotted exist in the virtual 3D world. Thissimplified illustration omits the person's arms and hands that may beholding the movable display device MD.

In the example non-limiting implementation, the real-world attitude ofdisplay device MD determines the virtual world viewpoint, viewingperspective or direction, or other viewing transformation for generatingthe image that display device MD displays. For example, FIG. 2A showsmovable display MD displaying a perspective view of octahedron P3 withthe octrahedron displayed near the bottom of the movable display MDviewing frame. Moving the display device MD downward in the y direction(see FIG. 2B) causes the image to shift to displaying the bottom part ofthe octrahedron near the top of the display device viewing frame. Thus,the change of spatiality of movable display device MD changes theviewpoint, viewing perspective or other viewing transformation into thevirtual world used by system S to generate and present the display onthe movable display device.

FIGS. 3A and 3B show the same situation but with stationary display SDnow added. In one implementation, the viewpoint, viewing perspective ordirection, or other viewing transformation into the 3D virtual worldused to generate images for display by stationary display SD remain thesame. In the example non-limiting implementation, the person is seeingboth the image presented on movable display device MD and the imagepresented on stationary display device SD at the same time.Additionally, system S maintains at least some aspects of spatialcorrelation between the two displayed images as the person changes theorientation of the movable displayed image relative to the stationarydisplayed image. Thus, the person perceives that the portion of thevirtual world displayed on movable display device MD exhibits spatialityrelative to the part of the virtual world displayed on stationarydisplay SD. In particular, the change of spatiality is sensed by sensorsT and graphics source(s) G controls or enables movable display device MDto display a part of the octrahedron that intuitively corresponds tothis spatiality change. The effect is remarkable, highly intuitive,immersive and interactive. Especially if stationary display device SD isrelatively large (e.g., a modern large screen LCD, plasma or projectiontype display) and the 3D graphics generation employed is close tophotorealistic, the person accepts the image the stationary displaydevice SD displays as a virtual reality and accepts the image displayedby movable display MD as a movable patch or piece of that same virtualreality. The patch that movable display MD displays is multi-dimensionalin the sense that at least from the user's perception, system Smaintains spatial correlation between the patch and the image thestationary display SD displays no matter how the person moves themovable display.

The image the stationary display device SD displays can itself beinteractive and animated (e.g., live action), adding to the sense ofrealism. The patch displayed on movable display MD can also beinteractive and animated (e.g., live action). While the person may feelimmersed to some degree in a large stationary display presentation (andimmersion can be improved by providing 3D effects and interactivity),immersiveness of the stationary display device SD is substantiallyenhanced through the addition of a spatially-correlated image patchdisplayed on the movable display MD.

Movable display device MD is closer to the person, so it can also bequite immersive despite its relatively small yet adequate size. It mayalso display animated interactive images responsive to user input. Inone example embodiment, the person can control the viewpoint, viewingperspective, viewing direction or other viewing transformation ofmovable display device MD into the virtual 3D world by moving themovable display device. Moving display device MD provides an interactiveview into the virtual world that changes in view as the person moves herbody, head, etc. This functionality allows the person to enjoy a highdegree of immersive interactivity with the virtual 3D world—an effectthat is substantially enhanced by the omnipresence of stationary displaySD which provides a larger immersive image orienting the user within thesame virtual world. In one example implementation, the person has herown private porthole into the animated virtual 3D world without the needto wear restrictive gear such as a head mounted display or specialglasses.

Furthermore, the user in some applications can directly interact withthe patch the movable display MD displays. As one example, system S cansimulate the user touching parts of the virtual world by placing afinger or stylus on the movable display MD. See FIGS. 8B, 8E.

For example, moving display device MD from the attitude shown in FIG. 3Ato the attitude shown in FIG. 3B changes the content of the image itdisplays relative to the image the stationary display device SDcontinues to display. In both cases, the movable display device MD maydisplay what appears to be a subset of the part of the virtual worlddisplayed by stationary display device SD. However, the particularsubset changes depending on where the person moves movable displaydevice MD. In some example implementations, moving the movable displaydevice MD downward changes the virtual camera viewpoint, viewingperspective, viewing angle or viewing transformation used to generatethe image displayed on movable display device MD so that the bottomrather than the top of the octahedron displayed on the stationarydisplay SD is now displayed on the movable display MD. In otherimplementations, movable display MD may selectively display a subset ofthe virtual space or it may instead or in addition display otherinformation (e.g., menus, scores, statistics, supplemental text, orother auxiliary information) e.g., depending on the attitude of themovable display relative to the stationary display SD.

Additionally, during such usage in some non-limiting embodiments,movable display MD may act as a pointing device to specify the positionof a pointing object the stationary display SD displays. If a ray normalto the movable display MD intersects an assumed location of thestationary display SD, system S may use aspects of the attitude ofmovable display MD to control the position of a pointing object such asa cursor, site or other indicator on the stationary display SD.

As an additional example, FIGS. 4A and 4B show what may happen when theperson rotates movable display device MD about the roll axis. Before andafter the rotation, the person sees the octrahedron displayed on themovable display device MD in the same orientation as if the person wereable to view the virtual 3D world. In this particular example, when theperson's viewpoint through movable display device MD would intersect thepart of the virtual space the stationary display SD displays, themovable display seems to act as a transparent window into the virtualworld—although in some implementations the image the movable displaydevice displays can be enhanced or augmented in some way. Rotatingmovable display device MD in the pitch or roll directions could in someapplications change the 3D perspective of the image the movable displaypresents so that from the user's eye perspective the illusion of atransparent window is maintained. Further rotating the movable displayMD so the user would expect the window to expose perspectives or objectsnot currently displayed on the stationary display SD can expose suchadditional objects or perspectives in spatial consistency with theattitude of the movable display device. These objects can be still oranimated or a combination.

FIGS. 5A and 5B illustrate that in one example implementation, this isthe same octrahedron orientation displayed on the stationary display SD.Such display functionality thus preserves the illusion that the movabledisplay MD is a personal porthole or window through which the person canpeer into the virtual world. In some implementations, rather thanduplicate the view of part of stationary display SD on the movabledisplay MD, the movable display can be used to display other information(e.g., text, graphics, instructions, input fields, etc.) whenever theview of the virtual world that movable display MD would otherwisedisplay depending on its spatiality is coextensive with a view thestationary display SD already is presenting. In still otherimplementations, the other information displayed by movable display MDmay be an augmentation overlaid onto a view of the virtual world themovable display presents based on at least some aspects of its attitude.

FIG. 6 shows an example of moving the movable display device MD to anattitude such that the image it displays is not a subset of orcoextensive with the image the stationary display device SD displays. Inthis case, a ray normal to the plane of movable display MD does notintersect the assumed or actual location of stationary display SD. Inthe example shown, the stationary display device SD displays thepyramid, octahedron and prism (P2, P3, P4) but the cube (P1) is out ofview. By moving the movable display device MD to a spatial attitude thatinterposes the movable display device as a virtual window between theperson and the virtual cube, the person can “see” the cube on themovable display device. Because the person can see both displays SD, MDat the same time, movable display device MD enhances the spatiality ofthe immersive 3D virtual world by allowing the person to feel as if heis within and surrounded by the virtual world. The movable displaydevice MD allows the person to discover aspects of the virtual worldthat stationary display device SD is not exposing to view. For example,the person can direct the movable display device MD downward tovisualize the ground or other virtual surface the person is standing on(e.g., a green golf fairway, desert sand, the deck of a ship orspacecraft, or other virtual surface). See FIG. 7 for a golf examplethat shows the virtual golf ball and golf club head. Or the person coulddirect movable display MD upward to view the virtual ceiling, sky,stars, moon, or other virtual overhead space. The possibilities arelimitless. In such cases, extended spatial cues can be displayed on thestationary display SD if desired to help the person perceptuallycorrelate the images or action displayed on the movable display MD withthe images or action displayed on the stationary display SD. The movabledisplay MD in this instance acts as a spatial extension of thestationary display SD to make the system S multidimensionally spatialand spatially coherent and expand the user's field of view of thevirtual space.

There could be more than one stationary display SD and more than onemovable display MD. FIG. 8D shows an example with one stationary displaySD and two movable displays MD1, MD2. One user can hold one movabledisplay MD1, and another user can hold an additional movable displayMD2. Both users can see the same stationary display SD. This applicationis useful for situations where two (or more) users wish to share andinteract with the same virtual environment at the same time. The usersdo not need to be co-located. For example, system S could be distributedby network such as the Internet or other network, each user could haveher own stationary display SD and movable display MD, and all of thedisplays could display portions of the same virtual environment.

In other example implementations, there may be no stationary screen.Imagine N mobile screens that start with an initial calibration step tocorrelate their physical spatial orientation. After this calibrationstep, the N screens represent a physically spatially coherent view ofthe virtual database. This can even be extended to position correlationif there is a good way to sense positions unambiguously in allconditions. With some example implementations, orientation is determinedby using the magnetic sensor to correct orientational drift. However, itis possible to provide the same if a better positional sensor(s) existto have perfect location info.

Example Applications That Context-Switch Movable Display Based onAttitude

In some non-limiting applications, it may be desirable to use movabledisplay MD for different purposes or contexts depending on the attitudeand/or pointing direction of the movable display e.g., relative to thestationary display SD. FIG. 6A shows three different general types ofrelationships between the attitude of movable display MD and thestationary display SD:

(a) in range,

(b) out of range, and

(c) transition area.

FIG. 6A shows a number of out-of-range cases OR1-OR4 where the vectorprojecting from movable display MD that is normal to the plane of themovable display does not indicate or intersect a position within an areacorresponding to an ideal or assumed stationary display SD. FIG. 6A alsoshows an in-range case IR where the normal vector projecting frommovable display MD does indicate, intersect or point to a positionwithin an area corresponding to an ideal or assumed stationary display.FIG. 6A also shows a transition area TA surrounding or at borders orperipheries of the in-range case IR in which the normal projecting fromthe movable display may indicate or intersect an edge or periphery of anarea corresponding to an ideal or assumed stationary display.

In some example non-limiting implementations, system S can automaticallyswitch usage or context of movable display MD depending on whether themovable display's attitude or pointing direction is in-range orout-of-range. In one example non-limiting implementation, the pointingdirection of movable display MD may for example be defined by a raynormal to the plane of the movable display and extending outward fromthe back surface of the movable display away from the user. Such anexample non-limiting pointing direction thus corresponds to thedirection of the user's gaze if the user were able to look throughmovable display MD as if it were a pane of transparent glass.

In Range

FIG. 6B shows an example in-range case where the normal of movabledisplay MD indicates or intersects a position inside the area determinedvirtually according to the ideal attitude and extent of stationarydisplay SD. In this particular example, the actual attitude and extentof the stationary display SD may not be precisely known by system S (forexample, if the stationary display comprises a home television set, theuser may be using any of a variety of different sizes and configurationsof televisions), and so system S may make certain assumptions concerningthe size of the stationary display, the distance between the movabledisplay MD and the stationary display and the orientation of thestationary display SD relative to the movable display. These assumptionsare possible because most users typically at least generally face towardthe stationary display SD, usually position themselves more or less infront of the stationary display, and are unlikely to be situated veryclose or very far from the stationary display. Additional input by theuser (e.g., size or type of television screen) or otherwise may be usedto supplement these assumptions.

In such an in-range case for some non-limiting implementations, theposition of a pointer such as cursor or other object displayed on thestationary display SD (as shown in FIG. 6C) may be controlled by theattitude of the movable display MD. Moving the movable display MD to adifferent attitude will in such a non-limiting embodiment cause theposition of the pointer object displayed on stationary display SD tochange (e.g., based on information from the sensors T), such that themovable display functions as a free space pointing device.Simultaneously with changing the attitude of movable display MD, a usermay provide additional inputs (e.g., by touching a touch screen TS,depressing buttons, etc.) that may control aspects of images displayedby either or both displays MD, SD. Thus, in one implementation, movabledisplay MD acts as a virtual pointer for stationary display SD whenpointing at the stationary display, and displays spatially-correlatedadditional views of the virtual world responsive to aspects of themovable display's attitude when not pointing at the stationary display.

In such an example, a virtual camera defining the view for display bystationary display SD does not necessarily move or change according tothe attitude of movable display MD. Users will mainly see the stationarydisplay SD in many or most applications, so the virtual space thestationary display SD displays does not necessarily need to be displayedon the movable display MD. In such a non-limiting instance, the movabledisplay MD can be used to display other types of information such as forexample score (for a game), statistics, menu selections, user inputoptions, or any of a variety of other ancillary or other informationuseful to the user. In some cases, the movable display may overlay suchinformation to augment a displayed image of the virtual world from adirection based on the attitude of the movable display MD. Thus, in someimplementations, the movable display MD can display more limited (ordifferent, supplemental, control or non-redundant) information while theuser's attention is likely to be focused on the stationary display SD.

FIG. 6D show a bird's eye map or view of the virtual space displayed bystationary display SD. FIG. 6D shows with an arrow the direction of thevirtual camera for defining the image the stationary display SD displaysfrom a first person viewpoint of a virtual character within the virtualspace. Moving the virtual character within the virtual space may changethe view that stationary display

SD displays. Of course, in other implementations, it would be possibleto display the bird's eye or other view on movable display MD to permitthe user to more easily navigate through the virtual space displayed bythe stationary display SD.

Out-Of-Range

FIG. 6E shows an example non-limiting out-of-range case where the normalof movable display MD does not indicate, point to or intersect aposition inside the area determined virtually according to the idealattitude of stationary display SD. When the normal of the movabledisplay MD indicates the position outside the stationary display SDarea, system S may use the movable display MD to display the virtualspace and the direction of the virtual camera for generating imagesdisplayed by movable display MD may be controlled by the movabledisplay's attitude. See FIGS. 6 & 6F which show movable display MDdisplaying the virtual space from a direction that depends on themovable display's attitude. Meanwhile, in such a non-limiting case, thedirection of a virtual camera for creating images to be displayed by thestationary display SD is not necessarily affected by the attitude of themovable display MD. Users can see right, left upper, under and rear sideof the virtual space by moving the movable display MD, and can see thefront side of the virtual space by viewing the stationary display SD. Inthis non-limiting application, users can feel that their visibility hasbeen spatially expanded by being able to simultaneously view side andfront views of the virtual space.

In some non-limiting implementations, the pointing position of apointer, cursor or other object displayed on the movable display MD canalso be controlled, for example, to stay in the center of the movabledisplay device's display screen as the image beneath it changes inviewing direction to reflect the attitude of the movable display. SeeFIG. 6G. FIG. 6H shows a bird's eye view of an example virtual spaceshowing the direction of the virtual camera for the movable display MDshowing two viewing directions, one for the stationary display andanother for the movable display. Simultaneously displaying an image ofthe virtual space on the stationary display SD and an additional imagefrom a different viewing direction of the virtual space on the movabledisplay MD provides a perception of depth and spatial immersiveness forthe user. See also FIG. 12A.

Transition Area Case

FIGS. 6I and 6J show a transition area case that exists between thein-range and out-of-range cases described above. FIG. 6I shows that sometransition area TA can be configured between in-range and out-of-range(inside and outside) so that users can change their gaze (the directionof their eyes) from the stationary display SD to the movable display MDsmoothly by some processing. For example, when the attitude of movabledisplay MD corresponds to the transition area, the image for thestationary display SD may provide some visual, audible and/or tactileeffect (e.g., the stationary display SD may scroll slightly) so thatusers can know when the image of the movable display MD changes from thein-range condition to the out-of-range condition or vice-versa. In oneexample implementation, a slight scrolling or other movement of thestationary display SD responsive to the relative attitudes of themovable and stationary displays MD, SD can be used to alert the userthat a context change will occur if the user continues to rotate orotherwise change the attitude of the movable display MD. The same ordifferent image or other effects can be used indicate transition fromin-range to out-of-range and out-of-range to in-range.

Thus, if the user is aiming at the big screen of stationary display SDusing the movable device MD, he is likely always watching the big screenand controlling the target. If the user targets near the edge area, thetransition mode may exist. The big screen slightly or marginally shiftstoward the direction of the gun direction. But after some marginalmovement, suddenly the smaller screen of movable display MD startsdisplaying portions of the virtual space outside the stationary displaySD. This can be a dramatic effect. At the transition, a slight lookaboutfor a small degree is possible to move the image displayed by thestationary display SD. Such a marginal or peripheral lookabout shift(e.g., based on the presence of a cursor or other pointer objectcontrolled in accordance with aspects of the attitude of the movabledisplay) can be provided in the direction of and in response to movementof movable display MD, e.g. to alert the user that a context switch ofthe movable display is about to occur either from a lookaround displaymode to an auxiliary information display mode or vice versa.

More Detailed Non-Limiting Examples

As shown in FIG. 8A, display device MD is structured and configured tobe easily holdable and movable by a person. For example, display deviceMD may be relatively small, portable and lightweight so it can be heldin, gripped and supported by, manipulated by and movable by the person'shand(s). In the particular example shown in FIG. 8A, display device MDis configured and structured to be held in the person's left and righthands simultaneously. With the person's two hands holding and supportingthe movable display device MD, the thumb of the person's left hand canoperate some manual input controls (e.g., a left-hand slide pad 50and/or a D-pad switch 52) while the thumb of the person's right hand cansimultaneously operate other manual input controls (e.g., a right-handslide pad 54 and/or a button array 56). There are additional controls 58a, 58 b that can be operated by the person's pointer fingers, and morecontrols such as triggers 60 on the rear surface (see FIG. 10B, 10D)that can be operated by the person's middle finger for example while theperson is holding display device MD. More detail concerning the detailedelectromechanical structure and operation of movable display device MD(and see also FIGS. 10A-10D showing an example non-limiting ergonomichandheld design) may be found in U.S. patent applications Ser. Nos.13/019924 and 13/019928 filed on Feb. 2, 2011, the contents of which areincorporated herein by reference in their entirety. In the FIGS. of10A-10C, the movable or handheld device MD housing 200 includes a frontplanar surface 202 and a rear substantially non-planar surface 204. Ascan be seen in FIGS. 10B, 10C, the rear non-planar surface 204 includesa sloping curved portion 206 that slopes up to a longitudinal ridge 206.A depression 210 on the other side of ridge 206 may provide variousreceptacles 212 for accepting accessories such as legs for a stand (seeFIG. 8C) or other peripheral devices. Finger-depressible buttons 60 a,60 b provided on sloping curved portion 206 (see FIG. 10D) are curved tohave a profile that matches the profile of the sloped curved portionwhile extending outward to provide comfortable resting pads for theuser's middle digits. Thus, the user can operate the buttons 60 withmiddle fingers and operate trigger buttons 58 with pointer fingers.Other devices, sensors, connectors, receiving receptacles and otherstructures (some of which may be useful for mating the movable displayMD with other devices) may be disposed on peripheral surfaces 212, 214.The implementation is not limited to this form factor; other formfactors (e.g., tablet computer, IPAD, smart phone, ebook reader, PDA,other) are possible for movable handheld display device MD.

In some example implementations, display device MD could be worn by aperson, for example attached to a part of the person's body such as theperson's wrist, forearm, upper arm, leg, foot, waist, head, or any otherportion of the person's body or clothing. In other exampleimplementations, movable handholdable display device MD can be fixedlyor otherwise mounted to a movable or fixed inanimate structure (seee.g., 7, 8C). For example, in some example arrangements, as shown inFIG. 8C, display device MD can be temporarily fixed to or stood on anon-movable structure (e.g., using an integral pull-out or other standthat maintains an easily viewable attitude of display device MD relativeto another structure such as a tabletop, a floor, a frame, etc.). Inother example arrangements, even though display device MD may betemporarily fixed in attitude relative to another structure, a personcan reorient it as desired or pick it up and move it around.

As mentioned above, non-limiting display device MD has a touch screen TSthat can be controlled by touching it with a stylus (see FIG. 8E) or afinger (see FIG. 8B). The touch screen can be used to provide variousinputs including handwriting, gestures, etc.

Sensors T determine the attitude of display device MD. Sensors T may becontained within display device MD, placed outside of display device MD,or some sensors may be disposed within display device MD and othersensors may be disposed externally to display device MD. More detailconcerning example non-limiting arrangements for sensors T may be foundin U.S. patent applications Ser. Nos. 13/019,924 and 13/019,928, filedon Feb. 2, 2011, the contents of which are incorporated herein byreference in their entirety. Special solutions or applications may use G(gravity) only, M (magnetic) only, or MG (magnetic+gyro) only out of“MARG” for 2D, leaner or reduced movement or special games or otherapplications.

In one particular example, sensors T may comprise a “MARG” sensor arrayspread between display device MD and a movable accessory device to whichthe movable display MD can be fixedly attached (see FIG. 9A, 9B). Forexample, the magnetometer of MARG can be disposed within the accessorydevice, and the accelerometer and gyrosensors of MARG can be disposedwithin the movable handheld display device MD housing. When the movabledisplay device MD and the accessory device are mated, electricallyconnections can be established that allows the magnetometer within theaccessory device to receive power from the movable handholdable displaydevice MD and for the movable display device to receive data from themagnetometer. Other variations are possible.

In other implementations, sensors T shown in FIG. 1 may include anydesired sensor(s) such as mechanical, acoustic, ultrasonic, camera,optical, infrared, magnetic, electromagnetic (e.g., radio or microwave),inertial, gyroscopic, acceleration-based, fiducial-based,outside-looking-in, inside-looking-out, GPS, cell phone, and/or anycombination thereof.

As mentioned above, in some example non-limiting implementations,sensors T representing a collection of sensors which makes up MARG maybe distributed in one or more housings or places. Such housings can befor example either stationary or moved spatially by the user. The subsetof housing elements can be combined together to behave as a singleelement as shown in FIGS. 9A, 9B.

In one example, sensors T may include some or all of the sensors used inthe Nintendo Wii Remote Controller and/or Nintendo Wii Remote Plus videogame controller, i.e., a direct pointing device, a triaxial or otheraccelerometer, and/or a triaxial or other gyroscope. Such sensors Tcould also include a multi-axis single-chip magnetometer, aninside-looking-out or outside-looking-in external optical, ultrasonicand/or other tracker, or many other variations as would be understood bythose skilled in the art. Although not necessary for many applicationssuch as home virtual reality and video game play, it might be desirablein certain applications to accurately track the complete pose (positionand orientation) of the movable display MD with a desired degree ofaccuracy to provide nearly complete fidelity in spatial correlationbetween the image displayed by the movable display MD and the image thestationary display SD displays. The items listed below and incorporatedby reference enable one skilled in the art to implement any desiredarrangement of sensors T based on particular system requirements anddesired applications:

-   -   X. Yun et al, “Design and Implementation of the MARG Human Body        Motion Tracking System,” Proceedings of the IEEE/RSJ        International Conference on Intelligent Robots and Systems (IROS        04), Sendai, Japan, September-October 2004;    -   E. R. Bachmann et al, “Inertial and Magnetic Posture Tracking        for Inserting Humans into Networked Virtual Environments,” ACM        Symposium on Virtual Reality Software and Technology, VRST 01,        Banff, Alberta, Canada, November 2001, pp. 9-16;    -   Bachmann, “Orientation Tracking for Humans and Robots Using        Inertial Sensors”, Computational Intelligence in Robotics and        Automation Proceedings (IEEE 1999) (this reference discloses        orientation and position tracking for human movement);    -   Eric Bachmann, “Inertial and Magnetic Angle Tracking of Human        Limb Segments for Inserting Humans into Synthetic Environments,”        PhD in Computer Science, Naval Postgraduate School, December        2000;    -   Welch et al, “Motion Tracking: No Silver Bullet, but a        Respectable Arsenal”, IEEE CG&A (November/December 2002)    -   Wormell et al, “Advancements in 3D Interactive Devices for        Virtual Environments” (2003);    -   U.S. patent application Ser. Nos. 13/019924 and 13/019928, filed        on Feb. 2, 2011;    -   U.S. Pat. No. 7,942,745;    -   U.S. Pat. No. 7,931,535;    -   U.S. Pat. No. 7,927,216;    -   U.S. Pat. No. 7,925,467;    -   U.S. Pat. No. 7,924,264; and    -   U.S. Pat. No. 7,920,985.        The items listed above are incorporated by reference for        purposes of enabling and disclosing a variety of alternative        ways to determine attitude and other characteristics of movable        display MD in multi-dimensional space.        Example Block Diagram

FIG. 11 shows an example non-limiting block diagram of movable displayMD (for more details see U.S. patent application Ser. Nos. 13/019,924and 13/019,928 filed on Feb. 2, 2011, the contents of which areincorporated herein by reference in their entirety). Manual controlinputs from the manual controls detailed above are supplied to processor102. The processor may harvest data inputs (including from touch paneldisplay TS) and report them via a radio transceiver 106 or otherwireless or wired connection to a remote console or computer G foranalysis. The remote console or computer G (which may also be connectedto stationary display SD wirelessly or by wire) may process the reporteddata inputs and generate compressed or other images that it may send tothe movable display radio transceiver 106. The movable display deviceprocessor 102 may decompress the received images and display them ontouch panel display TS.

The remote console or computer G can be one or more graphics generatorslocated in one place or distributed in a variety of places communicatingvia one or more networks. Such graphics generator(s) can useconventional 3D graphics transformations, virtual camera and othertechniques to provide appropriately spatially-coherent or other imagesfor display by the displays MD, SD. For example, the graphics generatorG can be any of:

a graphics generator that is part of or is a separate componentco-located with stationary display SD and communicates remotely (e.g.,wirelessly) with the movable display MD; or

a graphics generator that is part of or is a separate componentco-located with movable display MD and communicates remotely (e.g.,wirelessly) with the stationary display SD or associated equipment; or

a distributed graphics generating arrangement some of which is containedwithin the movable display MD housing and some of which is co-locatedwith the stationary display SD, the distributed portions communicatingtogether via a connection such as a wireless or wired network; or

a graphics generator located remotely (e.g., in the cloud) from both thestationary and movable displays SD, MD and communicating with each ofthem via one or more network connections; or

any combination or variation of the above.

In the case of a distributed graphics generator architecture orarrangement, appropriate data exchange and transmission protocols areused to provide low latency and maintain interactivity, as will beunderstood by those skilled in the art.

In one particular example, the MARG attitude sensors (i.e., a triaxialaccelerometer, a triaxial gyroscope and a tri-axial magnetometer)provide sensor outputs to a processor 102 that executes instructionsstored in non-transitory firmware storage 104. An example non-limitingmethod uses the 3-axis gyroscope (Angular Rate) with error correction byusing 3-axis accelerometer based Gravity vector reference (for roll andpitch), and 3-axis Magnetometer based north vector reference (for yaw).This so called orientation or attitude measurement is by “MagneticAngular Rate Gravity (MARG)”. In the example shown, the MARG sensors Tare not capable of detecting absolute position and use a magnetometer todetect what can be thought of as something like magnetic compass headingrelative to the earth's magnetic field.

It is not necessary in many applications for system S to track theabsolute or even relative position of movable display MD in 3D spacebecause the user's depth perception does not in at least some suchapplications require such fidelity. For example, the human eye includesa lens that changes shape to focus on near and far objects, and the useruses monocular and binocular cues to perceive distance. See e.g.,Schwartz et al, “Visual Perception, Fourth Edition : A ClinicalOrientation” Chapter 10 (McGraw Hill 2004). In many applications,movable display MD will fill a large part of the user's near vision, andstationary display SD will be in only a portion of the user's farvision. Thus, in many applications, the user's eye may focus on one ofthe two displays at a time. The user may first look at one display andthen at the other even though both displays are both in the user's fieldof vision. This may mean that in many useful applications, the fidelityof the positional spatial correlation and coherence does not need to beprecise in order to create the desired immersive user interface effect,and maintaining precise positional or distance spatial correlationbetween the two displays may be unnecessary. In other applications,other arrangements may be desired.

Additionally, using practical consumer grade components, there may beinstances where system S is unable to keep up with rapid movement ofmovable display MD. Luckily, during such rapid movement the personholding the movable display MD will also generally not be able to seethe image it's displaying. Thus, it may be generally sufficient in manyapplications for system S to maintain spatial correlation for movabledisplay MD for relatively static or slow-moving situations when the usercan see the image the movable display MD displays.

An initial calibration step can be used to allow system S to establishand maintain attitudinal spatial coherence between the two displays SD,MD. In such case, the person may be asked to orient the movable displayMD in a certain way relative to the stationary display SD to calibratethe system so that the magnetic compass bearing (e.g., NNW or 337.5°) ofstationary display SD within the room or other environment is known.Calibration of movable display MD is possible in a general free space(3D) orientation, or transformation (movement) case (note that somespecial usage like 2D or linear orientation or movement as the subset offree space usage would also be possible). There, the examplenon-limiting implementation relies on the relative movement of themovable display MD with initial calibration.

To calibrate during setup, the person may hold the screen of movabledisplay MD parallel to the stationary display SD while generally aiminga perpendicular vector at the center of the stationary display. A buttonor other user control may be pushed. The system S will measure and storethe magnetic field in the room to determine the orientation that themovable display MD is in when the user says it is pointing at thestationary display SD. This calibration assumes (1) the magnetic fieldof the room doesn't change, (2) the stationary display SD isn't moved,and (3) the movable display MD is set up with this stationary display SDin this room. If the magnetic field is changed or the stationary displaySD is moved or the system S is set up in another location, recalibrationmay be desirable. For example, recalibration may be performed when theroom furniture or a metal panel in the room moves, or the battery isreplaced (metal difference of alkaline batteries), or the environmentchanges the magnetic field in the room. In some residential areaslocated near street cars or light rails, dynamically changing magneticfields may exist. It is possible to filter unexpected dynamicdisturbance of the earth magnetic field by the initial magnitude ofmagnetic readouts, or unexpected values against gyroscopecorrespondence.

FIG. 12 is a flowchart of software instructions at least portions ofwhich are stored in non-transitory storage device 104 of movable displayMD and/or within a non-transitory storage device associated with remotecomputer G (for more details see U.S. patent application Ser. Nos.13/019,924 and 13/019,928 filed on Feb. 2, 2011, the contents of whichare incorporated herein by reference in their entirety). In the exampleshown, Viewpoint/Frustum of Virtual 3D World For Stationary Display (SD)is defined (block 120) (e.g., based on inputs from user controls on themovable display MD and/or on other user input devices such as the WiiRemote Plus and Nunchuk shown in FIG. 8A, 8B). Sensors T then acquirethe attitude of movable display MD (block 122), and a Viewpoint/Frustumof Virtual 3D World For Movable Display (MD) is defined (block 124).Then, system S uses conventional high speed interactive computergraphics hardware and software to project a 3D World onto the stationarydisplay (SD) based on the viewpoint/frustum defined for the stationarydisplay (block 126), and the 3D world is also projected onto the movabledisplay (MD) based on the viewpoint/frustum defined for the movabledisplay (block 128). Block 128 can use some or all aspects of theacquired attitude of movable display MD to change the transformation(s)used to project the 3D virtual space onto the movable display MD. Suchuse of acquired attitude may be in any desired way such as by alteringviewpoint, viewing angle, field of view, scale, rotation, viewingfrustum, type of projection, texture mapping, special effects or anyother aspect of the graphics processing and calculation(s) used toobtain a display view from the 3D virtual space. As one non-limitingexample, block 128 can change the direction and/or orientation of viewbased on the determined orientation of movable display MD relative tothe earth's magnetic field. FIG. 12A illustrates a stationary displayviewing frustum and transformation SDT and a moving display viewingfrustum and transformation MDT. This non-limiting example shows thataspects of the moving display viewing frustum and transformation MDT(e.g., the rotational, positional and scaling aspects of thetransformation) used to generate the image to be displayed on themovable display MD may be determined by the current attitude of themovable display. Of course, the stationary display viewing frustum andtransformation SDT need not be fixed. In some applications, thisstationary display viewing frustum and/or transformation may change inresponse to user inputs, a software program, etc. In some exampleapplications, it may be desirable to use the same or substantially thesame viewpoint, location, neighborhood, region and/or vicinity withinthe 3D virtual world defining the two viewing frustums andtransformations. In this way, system S can present the user with twodifferent views of the 3D virtual space as might be seen from the sameor similar vantage point within the 3D virtual space. In terms of the“same” or similar location, some application may not use the exact samelocation for the two different views, but might instead for example usea slightly higher position to give a more overview to one of thescreens. Other possibilities for the same or similar location, region,neighborhood and/or vicinity include:

-   -   higher or lower position    -   forward position or further back    -   looking slightly up or down (pitch)    -   zoomed out or in.    -   slightly rolled or yawed    -   positioned slightly left or right    -   other differences.        In other applications, different viewpoints, location,        neighborhood, region and/or vicinity within the 3D virtual world        can be used to generate images for display by the stationary        display SD and movable display MD, respectively.

FIG. 12B shows example steps performed by program instructions stored innon-transitory memory to switch the movable display between contexts. Inone example non-limiting implementation, system S detects the attitudeof the movable display (block 150) and determines whether a vectornormal to the plane of the display surface or housing of movable displayMD intersects an imaginary surface that bounds the stationary displaySD. The imaginary surface may be a rectangle, bounding box, sphere,plane, or any other surface. If intersection is found (indicating therelative orientations or postures of the movable and stationary displaysMD, SD falls within a certain range and thus that the movable display ispointing at a region in space where the stationary display SD is likelyto be) (block 152), then system S infers that the user intends to usemovable display MD as a pointing device to point at (or otherwiseinteract with) stationary display SD (“in-range” or auxiliary displaycontext). In such case, the system S uses the movable display as apointing device (e.g., to control the position of a pointing symboldisplayed on the stationary display SD) and may also display auxiliaryinformation on the movable display (block 154). If no intersection isfound (“out-of-range” or lookabout context) (block 156), system Sinstead causes the movable display MD to present an image of the virtualworld from a viewing direction corresponding to the movable display'sattitude (block 156). In either case, if the intersection is within aborder or peripheral area (“transition area”) (block 160), then one orthe other or both of displays MD, SD may display a visual effect (e.g.,move or scroll the display) indicating this border, marginal orperipheral condition (block 160).

The FIG. 12, 12B process may continually repeat many times to produce asequence of e.g., animated images that update interactively in real timeas the user moves movable display MD in free space. In someimplementations, each of steps in these figures may be performed in theremote computer G. In other implementations, the steps are performed bythe processor 102 of movable display MD. In still other implementations,some or all of the steps may be performed by a processor within orassociated with the stationary display SD. In still other non-limitingimplementations, some or all of the steps may be performed by one ormore accessory devices such as shown in FIGS. 8A-8B.

As shown in FIGS. 9A and 9B, in some example arrangements the movabledisplay MD can be attached to an accessory such as a handheld plasticgun or other aiming aid. The gun G shown in FIG. 9A includes a CMOSsensor (e.g., of the type found within a conventional Nintendo WiiRemote Plus) for detecting a set-top sensor IR bar, which can be used asa fiducial or reference source for an absolute measurement. The CMOSsensor viewing frustum may be limited, but many or most “shooter” andpointing type applications are performed by facing towards thestationary display SD. These two optical (IR) set-top lighthouses orbeacons would help to simplify or even eliminate the initial calibrationor recalibrations. One example non-limiting more comprehensive usagewould be MARG+optical (e.g., CMOS sensor), which would provide acombination of inertial, optical and magnetic position and orientationsensing. Such outputs can be fused or combined e.g., using Kalmanfiltering or other techniques as described in Welch et al (cited above)to achieve a fused result. In the example movable display MD not usingthe accessories shown in FIG. 9A, 9B, there is no optical directpointing device, which would have been the only reference source for theabsolute measurement related to TV screen location. In suchapplications, system S provides attitudinal spatial correlation onlybetween the two displays MD, SD and does not provide any absolute orrelative positional spatial correlation.

While the technology herein has been described in connection withexemplary illustrative non-limiting embodiments, the invention is not tobe limited by the disclosure. For example, given the conceptual natureof the present disclosure and its detailed exploration of a variety ofdifferent possible implementations, many statements herein do notcorrespond to any particular actual product that may be eventually bemade available to consumers. Additionally, while the preferredembodiments use a stationary display SD and a movable display MD, bothdisplays could be movable or both could be stationary, or there could bemore displays some of which are movable and some of which arestationary. One or both the displays SD and MD may a 3D display (e.g.,an autostereoscopic display) that provide stereoscopic perception ofdepth to a viewer. One or both of the displays SD and MD may beso-called high-definition displays (e.g., 1,280×720 pixels (720 p) or1,920×1,080 pixels (1080 i/1080 p)). Still additionally, the imagesdisplayed on the displays SD and MD may be from any source such as ananimation engine, a video game machine, a simulator, or a video which isappropriately transformed or processed to provide displays fromdifferent perspectives. The invention is intended to be defined by theclaims and to cover all corresponding and equivalent arrangementswhether or not specifically disclosed herein.

1. A handheld for use with an image of a virtual 3D space displayed froma first 3D viewing direction on a stationary display to provide animmersive spatial human-machine interface, the handheld comprising: amoveable housing dimensioned to be grasped and supported by the left andright hands of a user at ends thereof, the housing being configured anddimensioned to move with the user's hands to change position andorientation in free space; a gyrosensor disposed within the housing, thegyrosensor sensing angular rates of the housing in three axes; anaccelerometer disposed within the housing, the accelerometer sensinglinear accelerations of the housing in three axes; a magnetometerdisposed within the housing, the magnetometer sensing direction of theearth's magnetic field relative to the housing; a wireless transceivingarrangement disposed within the housing and coupled to the gyrosensor,the accelerometer and the magnetometer, the wireless transceivingarrangement (a) wirelessly transmitting sensed angular rates, sensedlinear accelerations and sensed magnetic field direction and (b)wirelessly receiving a video signal representing moving images of thevirtual 3D space from a second 3D viewing direction that is responsiveto the transmitted sensed angular rates, sensed linear accelerations,and sensed magnetic field direction; and a display on the moveablehousing, the display coupled to the wireless transceiving arrangementand configured to display the moving images represented by the receivedvideo signal to provide spatial correlation between attitude of themoveable housing and images of the virtual 3D space displayed by thehandheld display, the handheld being structured to allow the user toview the virtual 3D space from the first 3D viewing direction on thestationary display while simultaneously viewing, on the handhelddisplay, the same virtual 3D space from the second 3D viewing directionspatially correlated with attitude of the moveable housing to therebyprovide immersive 3D spatiality.
 2. The handheld of claim 1, wherein animage for the handheld display represents at least a part of the virtual3D space which is not represented in an image for display by thestationary display.
 3. The handheld of claim 1, wherein the stationarydisplay displays images from a reference viewing direction, and thehandheld display displays images from a different, free or arbitraryviewing direction which alters according to determined attitude aspectsof the handheld, the different, free or arbitrary viewing directionbeing based on the reference direction.
 4. The handheld of claim 1wherein images the handheld display displays represent at least a partof the virtual 3D space which is not represented in an image displayedby the stationary display when determined attitude aspects of thehandheld are out of the range corresponding to the stationary display.5. The handheld of claim 1 wherein images the handheld display displaysrepresent at least a part of the virtual 3D space which is representedin an image displayed by the stationary display when determined attitudeaspects of the handheld are in a range corresponding to the stationarydisplay.
 6. The handheld of claim 1, wherein sensed handheld attitudedefines a virtual camera, and wherein the images displayed by thehandheld display alter the viewing direction of the virtual cameraaccording to determined attitude aspects of the handheld substantiallywithout altering the viewing direction of the virtual camera of thestationary display.
 7. The handheld of claim 6, wherein the position ofthe virtual camera for the handheld display and the position of thevirtual camera for the stationary display are substantially the same. 8.The handheld of claim 1 wherein the handheld display displays based on amodified projection viewpoint in response to determined attitude aspectsof the handheld display.
 9. The handheld of claim 1 wherein the handhelddisplay comprises a touch display.
 10. The handheld of claim 1 whereinthe handheld display displays a patch of the virtual 3D space that isspatially correlated with the image displayed by the stationary display.11. The handheld of claim 1 wherein the handheld display displays movingimages that are spatially correlated with the images the stationarydisplay displays.
 12. The handheld of claim 1 wherein the handheldorientation determines the position at which a pointing object isdisplayed on the stationary display when the handheld display ispointed/aimed at the stationary display, and the handheld displays animage of the virtual 3D space from a viewing direction responsive to atleast some aspects of attitude of the handheld when the determiningdetermines that the handheld display is not pointed/aimed at thestationary display.
 13. The interface of claim 1 wherein the handhelddisplay displays a patch of the virtual 3D space that is spatiallycorrelated with the image displayed by the stationary display.
 14. Adisplay device for use with a stationary display displaying an image ofa virtual 3D space from a first 3D viewing direction to provide animmersive spatial human-machine interface, the display devicecomprising: a housing graspable and supportable by the hands of a user,the housing being configured and dimensioned to move with the user'shands to change attitude in free space; a MARG sensor array disposedwithin the housing, MARG sensor array sensing attitude of the housing; awireless radio disposed within the housing and coupled to the MARGsensor array, the wireless radio (a) wirelessly transmitting sensedattitude and (b) wirelessly receiving a video signal generated inresponse to the transmitted sensed attitude; and a display on thehousing, the display coupled to the wireless radio and configured todisplay moving images of a part of a virtual 3D space responsive to thereceived video signal to provide spatial coherence between attitude ofthe housing and images of part of the virtual world displayed by thedisplay, the display device being structured to allow the user to viewthe virtual 3D space on the stationary display from the first 3D viewingdirection while simultaneously viewing, on the handheld display, thesame virtual 3D space from a second 3D viewing direction spatiallycorrelated with attitude of the housing to thereby provide immersive 3Dspatiality.
 15. The display device of claim 14, wherein an image for thehandheld display represents at least a part of the virtual 3D spacewhich is not represented in an image displayed by the stationarydisplay.
 16. The display device of claim 14, wherein the stationarydisplay displays images from a reference viewing direction, and thehandheld display displays images of a different, free or arbitraryviewing direction which alters according to the determined attitudeaspects of the handheld display, the different, free or arbitraryviewing direction being based on the reference direction.
 17. Thedisplay device claim 14 wherein images the handheld display displaysrepresent at least a part of the virtual 3D space which is notrepresented in an image displayed by the stationary display when thedetermined attitude aspects are out of the range corresponding to thestationary display.
 18. The display device of claim 14 wherein imagesthe handheld display displays represent at least a part of the virtual3D space which is represented in an image displayed by a stationarydisplay when determined attitude aspects of the display device are in arange corresponding to the stationary display.
 19. The display device ofclaim 14 wherein the handheld display displays based on a modifiedprojection viewpoint in response to the attitude of the display device.20. The interface of claim 14 wherein the handheld display comprises atouch display.